Conventionally, a server may receive many files from various entities. The server may be required to execute or perform actions on the files. The actions may include loading the files, copying the files, manipulating the files, executing the files and/or any other sort of action.
The files may be received on a schedule. At times, the files may arrive prior to scheduled arrival, on schedule, or after scheduled arrival.
In jobs that include file or data loading, there may be a need for a job to be triggered when a file arrives in a file or data landing zone. When the file arrives in a specified area or landing zone, an underlying system may utilize the file as input and process, or load the file. The execution of data or file load jobs may be triggered by scheduler tools such as Cron tab, Autosys or any other suitable scheduler tools. The scheduler tools enable the creation of a watcher script to recognize the incoming files.
There are numerous problems associated with creating watcher scripts within a scheduler tool. A first problem may be that, if a data load job requires multiple files as input, there may be a need to create multiple watcher scripts (each watcher script corresponding to a single file) in the scheduler tool. At times, there may be tens or hundreds of files per job at it may be time consuming and tedious to create multiple watcher scripts, maintain the scripts and analyze the scripts when there is an issue (such as, in a debugging scenario) for each of the files.
A second problem may be that, using the existing scheduler tools, a user is not informed about the historical information pertaining to previous file observation. The user or the system, therefore, cannot use the historical information to make educated predictions, in the future, relating to similar files.
A third problem may be when there is an issue, or input files are not available in the landing area or specified area, the scheduler tools may present an alarm. The alarm is typically a general alert that there is an issue but there is no specific point to remedy. Therefore, a user may be required to debug the entire script and/or file to attempt to remedy the issue point.
A fourth problem may be that multiple scripts may initiate multiple threads in the operating system and therefore, may occupy a large amount of system resources.
A fifth problem may be that the scheduler tools do not have the capability to validate a list of files against the files that have arrived and/or the files that have been loaded.
A sixth problem may be that the existing scheduler tools may not capture names for each file and therefore, it may be difficult to identify the root cause or root system that sent and incorrect file.
A seventh problem may be that the existing scheduler tools may be unable to validate calendar events relating to particular files.
As the amount of files and data being loaded multiplies, it becomes more and more difficult to maintain a system that includes the aforementioned problems. Furthermore, the inconsistency in file arrival may cause bottle necks at the server.
Therefore, performing predictive analysis on files that are scheduled to arrive may be desirable. Also, scheduling arrival times of files based on the predictive analysis may be desirable. Additionally, observing the actual arrival times of the files may be desirable. A system that combines performing predictive analysis, scheduling arrival times of files and observing actual arrival times of files may be further desirable. It would be yet further desirable to use this system in connection with loading multiple files.